Notes

This report was generated on 2017-11-27 12:16:49.

…

R-Script & data

The preprocessing and analysis of the data was conducted in the R project for statistical computing. The RMarkdown script used to generate this document and all the resulting data can be downloaded under this link. Through executing main.Rmd, the herein described process can be reproduced and this document can be generated. In the course of this, data from the folder ìnput will be processed and results will be written to output.

GitHub

The code for the herein described process can also be freely downloaded from https://github.com/mycid/rddj-template-TE-Raster-Extract.

The code for the project template may be found at https://github.com/grssnbchr/rddj-template.

Many thanks to Timo Grossenbacher @ (https://github.com/grssnbchr)

for creating this extremely elegant and useful template …

Project Purpose

This project was created to facilitate easy conversion of L2 netCDF rasters taken from NASA’s OceanColor Website run by the Ocean Biology Processing Group (OBPG) at NASA’s Goddard Space Flight Center (https://oceancolor.gsfc.nasa.gov/). Potential exists for others to easily adapt this script to handle L1 netCDF files as well or other large netCDF remote sensing files. The goals of the code are two-fold: 1. Convert many unprojected netCDF rasters to any raster file type in bulk. 2. Reproject and clip to the same study area many netCDF rasters with widely different swatch bounding boxes. An additional step has been to quickly visualize the results when necissary.

… ### Data description of input files Input files are netCDF files downloaded from the Ocean Color website as L2 products. These products are processed variables such as seasurface temperature and chlorophyll which are dervived from imperical relationships between irradiance and the physical variables. The l2 product is conveniently defined here: https://oceancolor.gsfc.nasa.gov/products/ netCDF files are structured to contain global attirbutes, band attributes and multiple bands. The metadata and file naming system is described here: https://oceancolor.gsfc.nasa.gov/docs/format/l2nc/ Example data for running this script includes 3 files containing chlorophyll alpha concentrations with associated variables and 3 files containing sea surface temperature with associated variables. Both files correspond to three images the study area contains on a single day, December 28, 2016.

Data description of output files

NASA_proj_metaD

Contains essential metadata extracted from the netcdf file global atributes. The bands extracted and the output raster file name are appended in columns at the end of the table. The metadata descriptive document may be found at: FILL IN

Output Raster files

Output raster files are generated and named using user specification of the function OCnetCDF2Raster. Raster files will be bounded by the specified study area and in the specified projection. Raster will not be scaled and cell values will contain the original values from the extracted raster. During rasterization and reprojection cells with multiple values from the netcdf file are averaged. NASA netCDF files are unprojected and use the WGS1984 latlon coordinate system. Each cell is given a centroid coordinate. Each raster has a resolution of aproximately 1km. This project uses UTM_S18 for the projection. Cells without data are NA. Users have the option of extracting the flagging layer contained within the netcdf file. For convenience, the flag values are described at the end of the document.

Output plots (optional)

Output plots can optionally be generated using the ExtractRaster or OCnetCDF2Raster function. Plots are saved as jpeg imaged and display the netCDF raster data in 10 discreet colored quantiles. The study area extent is also shown as a white polygon rectangle border. All points outside of the polygon are not included in the exported raster file but are shown here in the plot. Plots are titled by the date of the swath and saved with the same name as the original netcdf file included. …

Preparations

Define packages

# from https://mran.revolutionanalytics.com/web/packages/checkpoint/vignettes/using-checkpoint-with-knitr.html
# if you don't need a package, remove it from here (commenting is probably not sufficient)
# tidyverse: see https://blog.rstudio.org/2016/09/15/tidyverse-1-0-0/
cat("
library(raster)
library(ncdf4)
library(ggmap)
library(svMisc)
library(RColorBrewer)
library(tidyverse) # ggplot2, dplyr, tidyr, readr, purrr, tibble
library(stringr) # string manipulation
library(scales) # scales for ggplot2
library(jsonlite) # json
library(lintr) # code linting
library(sp) # spatial data handling
library(rgeos) # spatial data handling
library(rgdal) # spatial data handling",
file = "manifest.R")

Install packages

# if checkpoint is not yet installed, install it (for people using this
# system for the first time)
if (!require(checkpoint)) {
  if (!require(devtools)) {
    install.packages("devtools", repos = "http://cran.us.r-project.org")
    require(devtools)
  }
  devtools::install_github("checkpoint",
                           username = "RevolutionAnalytics",
                           ref = "v0.3.2", # could be adapted later,
                           # as of now (beginning of July 2017
                           # this is the current release on CRAN)
                           repos = "http://cran.us.r-project.org")
  require(checkpoint)
}
# nolint start
if (!dir.exists("~/.checkpoint")) {
  dir.create("~/.checkpoint")
}
# nolint end
# install packages for the specified CRAN snapshot date
checkpoint(snapshotDate = package_date,
           project = path_to_wd,
           verbose = T,
           scanForPackages = T,
           use.knitr = F)
rm(package_date)

Load packages

source("manifest.R")
unlink("manifest.R")
sessionInfo()
## R version 3.4.2 (2017-09-28)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 7 x64 (build 7601) Service Pack 1
## 
## Matrix products: default
## 
## locale:
## [1] LC_COLLATE=English_United States.1252 
## [2] LC_CTYPE=English_United States.1252   
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C                          
## [5] LC_TIME=English_United States.1252    
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] rgdal_1.2-7        rgeos_0.3-23       lintr_1.0.0       
##  [4] jsonlite_1.4       scales_0.4.1       stringr_1.2.0     
##  [7] dplyr_0.5.0        purrr_0.2.2.2      readr_1.1.1       
## [10] tidyr_0.6.3        tibble_1.3.3       tidyverse_1.1.1   
## [13] RColorBrewer_1.1-2 svMisc_0.9-70      ggmap_2.6.1       
## [16] ggplot2_2.2.1      ncdf4_1.16         raster_2.5-8      
## [19] sp_1.2-4           checkpoint_0.4.2  
## 
## loaded via a namespace (and not attached):
##  [1] reshape2_1.4.2    haven_1.0.0       lattice_0.20-35  
##  [4] colorspace_1.3-2  htmltools_0.3.6   yaml_2.1.14      
##  [7] rlang_0.1.1       foreign_0.8-69    DBI_0.6-1        
## [10] readxl_1.0.0      modelr_0.1.0      jpeg_0.1-8       
## [13] plyr_1.8.4        cellranger_1.1.0  munsell_0.4.3    
## [16] gtable_0.2.0      rvest_0.3.2       RgoogleMaps_1.4.1
## [19] mapproj_1.2-4     psych_1.7.5       evaluate_0.10.1  
## [22] knitr_1.17        forcats_0.2.0     rex_1.1.1        
## [25] parallel_3.4.2    broom_0.4.2       proto_1.0.0      
## [28] Rcpp_0.12.13      geosphere_1.5-5   backports_1.1.1  
## [31] mnormt_1.5-5      rjson_0.2.15      hms_0.3          
## [34] png_0.1-7         digest_0.6.12     stringi_1.1.5    
## [37] grid_3.4.2        rprojroot_1.2     tools_3.4.2      
## [40] magrittr_1.5      maps_3.1.1        lazyeval_0.2.0   
## [43] xml2_1.1.1        lubridate_1.6.0   assertthat_0.2.0 
## [46] rmarkdown_1.8     httr_1.2.1        R6_2.2.1         
## [49] nlme_3.1-131      compiler_3.4.2

Load additional scripts

# if you want to outsource logic to other script files, see README for 
# further information
knitr::read_chunk("scripts/Extract_Rasterize.R")
source("scripts/Extract_Rasterize.R")

Test if input data is present

paths <- list.dirs(path = "input/")
files <- list.files(path = paths, pattern = ".nc$", full.names = TRUE)
if (length(files) == 0){
  message("input netCDF files are not present in the input folder")
} else{
  message(paste(length(files), "files are present"))
}
## 6 files are present

Extract only metadata example

#Choose the band you wish to extract
band <- "chlor_a"
#name of netCDF file
nCDF <- files[1]
bnd <- paste("geophysical_data/", band, sep = "") # create full length band variable name
md <- metaXtract(nCDF, band = bnd)
print(md)
##                  title                product_name platform
## 1 HMODISA Level-2 Data A2016363183500.L2_LAC_OC.nc     Aqua
##                                        id             date_created
## 1 2014.0.1/L2/A2016363183500.L2_LAC_OC.nc 2017-01-12T21:37:07.000Z
##   spatialResolution geospatial_lat_max geospatial_lat_min
## 1              1 km          -43.70087          -59.26575
##   geospatial_lon_max geospatial_lon_min                    data1 data2
## 1          -59.34376          -81.40116 geophysical_data/chlor_a flags
##                                                file
## 1 input//netCDF_files/A2016363183500.L2_LAC_OC.x.nc

Extract from single file and generate plot

#Set additional variables
output.proj <- "+init=epsg:5362"
date <- NetCDFdateT(nCDF)
plotname <- paste(band, date[1])
proj.ext <-
  extent(250000,
         xmax = 1200000,
         ymin = 3700000,
         ymax = 5500000)
#
Rasta <- ExtractRaster(
  nCDF,
  band = band,
  Noflags = FALSE,
  output.proj = output.proj,
  proj.Ext = proj.ext,
  makeplots = TRUE,
  saveplots = TRUE,
  plotname = plotname
)
## [1] "input//netCDF_files/A2016363183500.L2_LAC_OC.x.nc"

####Check output folder to verify results

Use wrapper function to extract netCDF rasters from multiple files

#Set stage for use of the netCDF2Raster function
paths <- list.dirs(path = "input/")
files.oc <- list.files(path = paths, pattern = "OC\\.x\\.nc$", full.names = TRUE)
files.sst <-   list.files( pattern = "SST\\.x\\.nc$", full.names = TRUE)
 #list
chlor <- "chlor_a"
sst <- "sst"
proj.ext <-
  extent(250000,
         xmax = 1200000,
         ymin = 3700000,
         ymax = 5500000)
output.proj <- "+init=epsg:5362"

oc.netCDF.2raster(files.oc,
           chlor,
           filename = NA,
           output.extension = ".tiff",
           output.proj = output.proj,
           meta.name = "NASA_proj_metaD_Chlor.csv",
           proj.Ext =proj.ext,
           meta.exp = TRUE,
           Noflags = FALSE,
           makeplots = TRUE,
           saveplots = FALSE,
           makeMeta = TRUE)
## [1] "Extracting metadata..."
## Progress:  33%  Progress:  67%  Progress: 100%  Done!
## [1] "Extracting Rasters..."
## [1] "input//netCDF_files/A2016363183500.L2_LAC_OC.x.nc"

## Progress:  33%  [1] "input//netCDF_files/A2016363184000.L2_LAC_OC.x.nc"

## Progress:  67%  [1] "input//netCDF_files/A2016363201500.L2_LAC_OC.x.nc"

## Progress: 100%  Done!
##    user  system elapsed 
##  -93.63  -24.50 -149.03

Linting

The code in this RMarkdown is listed with the lintr package, which is based on the tidyverse style guide.

lintr::lint("main.Rmd")
## main.Rmd:100:64: style: Trailing whitespace is superfluous.
## path_to_wd <- csf()# if this - for some reason - does not work, 
##                                                                ^
## main.Rmd:200:1: style: lines should not be more than 80 characters.
## bnd <- paste("geophysical_data/", band, sep = "") # create full length band variable name
## ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
## main.Rmd:233:1: style: lines should not be more than 80 characters.
## files.oc <- list.files(path = paths, pattern = "OC\\.x\\.nc$", full.names = TRUE)
## ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
## main.Rmd:251:21: style: Put spaces around all infix operators.
##            proj.Ext =proj.ext,
##                     ^~

Flagging

Flags and meaning long_name: Level-2 Processing Flags valid_min: NA valid_max: 2147483647 flag_masks: 1 flag_masks: 2 flag_masks: 4 flag_masks: 8 flag_masks: 16 flag_masks: 32 flag_masks: 64 flag_masks: 128 flag_masks: 256 flag_masks: 512 flag_masks: 1024 flag_masks: 2048 flag_masks: 4096 flag_masks: 8192 flag_masks: 16384 flag_masks: 32768 flag_masks: 65536 flag_masks: 131072 flag_masks: 262144 flag_masks: 524288 flag_masks: 1048576 flag_masks: 2097152 flag_masks: 4194304 flag_masks: 8388608 flag_masks: 16777216 flag_masks: 33554432 flag_masks: 67108864 flag_masks: 134217728 flag_masks: 268435456 flag_masks: 536870912 flag_masks: 1073741824 flag_masks: NA flag_meanings: ATMFAIL LAND PRODWARN HIGLINT HILT HISATZEN COASTZ SPARE STRAYLIGHT CLDICE COCCOLITH TURBIDW HISOLZEN SPARE LOWLW CHLFAIL NAVWARN ABSAER SPARE MAXAERITER MODGLINT CHLWARN ATMWARN SPARE SEAICE NAVFAIL FILTER SPARE BOWTIEDEL HIPOL PRODFAIL SPARE